1. Field of the Invention
The present invention generally relates to the field of regression testing of complex systems. More particularly, the present invention relates to methods, systems, and media for generating an improved regression suite database for regression testing of computer systems, or components thereof.
2. Description of the Related Art
Verification of a complex integrated circuit system is an iterative process where the entire system (or at least all its major features) has to be tested on a continuous basis for the duration of the design. To alleviate the burden and to manage the increasing task of verification, regression testing is often adopted. Regression testing involves the repetitive testing of a design's major features to ensure changes and upgrades to the design have not introduced new bugs into the system. Regression test suites are necessary to ensure that changes to the system made as a result of previous tests (bugs found) or design upgrades/changes/enhancements have not “broken” something previously verified, or introduce a new bug somewhere else in the design. Examples of complex systems for which regression testing is used in the design include both hardware and software computer systems, such as integrated circuit (IC) chips.
Design of an IC chip, including its architecture, is a very complex, expensive, and time-consuming task, making verification of that design before fabrication critical. Verification of the design of a complex system such as an IC chip is an iterative process where the entire system (or at least all of its major features) is tested on a continuous basis for the duration of the design. As the design complexity increases, so does the state space and the number of functional states and finite state machines that need to be verified. In a typical microprocessor design environment, billions of simulation cycles are required to verify all features of the design.
The design process for an IC chip starts with the creation of a functional specification for the design. Once the functional specification has been completed, the verification team typically creates a test plan that specifies the functionality requiring testing at both the block and system levels. The team then creates testbenches (also known as tests) such as deterministic manually-written tests and automatically-generated tests to verify design functionality until the test plan requirements have been met. The process of verifying the functional specification of a design is called functional verification, which encompasses the development and implementation of a test plan.
Functional verification ensures functional conformance of a processor design to its architectural and microarchitectural specifications and determines whether the design is logically correct. The verification process involves developing and simulating tests that are used to determine whether design components (e.g., processor units, resources, functions, etc.) behave according to their functional specification, from both an architectural and microarchitectural perspective. Functional verification is desirably completed before fabrication of the processor, as finding and fixing errors, or bugs, after fabrication proves to be time-consuming and expensive.
When designing increasingly complex integrated circuits such as microprocessors, Application-Specific ICs (ASICs) and system-on-chips (SoC's), functional verification has proven to be a major bottleneck in achieving time-to-market goals. Design teams report that functional verification of medium- to large-complexity processors and ASICs may consume over 70% of the design team's manpower, schedule and budget. In spite of the time and resources consumed by functional verification, is an incomplete process, as design bugs are often not discovered until after fabrication.
Any situation with frequent application changes carries the risk of inadvertent problems being introduced, or regressed, into the application. To mitigate these risks, most organizations institute rigorous regression testing efforts. These initiatives however can become highly labor intensive and very expensive. Automatic Random Test Generation and simulation is sometimes used instead of test suite generation, maintenance and simulation. Such test generation environments run full-time and tests are simulated across a distributed simulation farm, resulting in a very large number of tests. Identifying the unique tests with the highest coverage is the goal of generating an optimized regression suite database.
Regression testing is a time-consuming and resource-intensive process used to validate a system's functionality following modifications. The cost-effectiveness of regression testing techniques varies with the characteristics of the test suites. One such characteristic, test suite granularity, involves the way in which test inputs are grouped into test cases within a test suite. Various cost-benefit tradeoffs have been attributed to choices of test suite granularity, but almost no research has formally examined these tradeoffs. It is not clear how this and similar regression testing techniques apply to a design with a very large number of inputs and state space.
Managing the ever increasing size of the regression suite database and the large simulation and verification resources required is a difficult problem. It is estimated that over 70% of the resources and development-cycle of a processor is used in its verification, with regression testing playing an important role. Due to increasing market pressure on the design turn around cycle, the functional specification, architectural definition and design and verification are conducted in parallel, or at least with large overlaps. Accordingly, regression generation and management in the field of processor verification is one of the most demanding and time/cost sensitive applications of the regression test suite concept.
Therefore, there is a need for methods, systems, and media to develop a minimal, high-coverage regression suite database that provides high verification coverage and reduces use of simulation time and resources.